Advancements in information technology and networks are transforming the everyday lives of many people with respect to employment, health care, communication, education, environment, etc. In particular, advancements in information technology and networks have spawned the field of Cyber-Physical Systems (CPSs), which field refers to the next generation of engineering systems integrated via advanced technologies and protocols. These engineering systems are capable of performing ubiquitous computing, communication, and control for complex physical systems and can be implemented in energy systems (e.g., the electric power distribution and smart grids), transportation systems (e.g., traffic networks), health care and medical systems, surveillance networks, control systems for underwater and unmanned aerial vehicles, etc. In many of these applications, sub-systems, sensors or observation posts, and controllers or control stations are distributed, often at distinct locations, and communication among sub-systems is limited. Thus, in the context of such systems, there is a demand for real-time communication, decentralized decisions, and the integration of real-time communication and decentralized decisions into complex networks.
In the field of communications, most encoders and decoders for transmitting information over channels (e.g., for transmitting speech signals over wireless communication channels) are designed based on an assumption that the channels do not have memory. That is, most communication systems are configured based on theories, methods, expressions, etc. assuming that channels of the communication system have an conditional output probability distribution that depends only on a current input (i.e., the output of the channels is conditionally independent of previous channel inputs or outputs symbols and the source symbols). However, typical communication channels are not memoryless due to Inter-symbol interference (ISI), correlated channel noise, etc. As a result, most communication systems are configured with components (e.g., encoders) that do not operate optimally when transmitting information over channels (e.g., the components do not achieve the capacity of the channels with memory) and are, in many cases, overly complicated due to the lack of knowledge of capacity achieving properties of the encoders. A characterization of channel capacity and corresponding capacity achieving channel input distributions, which would allow for the design of capacity achieving encoders, is not known for most channels with memory.
Further, the field of communications has developed primarily based on the following model: a message is generated randomly by an information source, the message is encoded by an encoder, the message is transmitted as a signal over a noisy channel, and the transmitted signal is decoded to produce an output as an approximation of the message generated by the source. The fundamental problem of this model is to determine simultaneously what information should be transmitted (source coding) and how the information should be transmitted (channel coding) to achieve performance. Over the years, this fundamental problem has been separated into the two subproblems of source coding and channel coding. The first sub-problem, source coding, is related to efficient representation of information, such as information representing speech, so as to minimize information storage and to characterization of a minimum rate of compressing the information generated by the source of the information (e.g., via the classical Rate Distortion Function (RDF) of the source subject to a fidelity of reconstruction). The second sub-problem, channel coding or “error correction coding,” is related to a correction of errors arising from channel noise, such as flaws in an information/data storage or transmission system, loss of information packets in networks, failures of communications links, etc., and to the characterization of the maximum rate of information transmission, called “channel capacity.”
The general separation of the fundamental problem into source coding and channel coding sub-problems, has divided the community of developers into independent groups developing source codes and channel codes, respectively. Although, extremely useful in some contexts, this idealized separation is limiting future advances in communication technology, in that developers are ignoring practical design criteria, such as computational complexity, delay, and optimal performance. Further, the ideal separation of source coding and channel coding is often violated for point-to-point communications over channels with memory and for network communication systems. On the other hand, the optimal design of simultaneously performing data compression and channel coding is, in those known cases, elegantly simple. However, this optimal design is, in general, hard to find. For example, separation of source and channel coding leads to the design of channel codes which treat all information bits as equally important. However, a scenario in which all information bits are equally important (e.g., in achieving an optimal channel capacity) is rare, and, hence, a separation of source and channel coding can lead to performance degradation.